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HI  Afc*tr#C» 

A  system  evaluation  method  is  presented  which  systematizes  and  quantifies  both 
PRP  pilot  rating  procedures  and  ECP  engineering  calculation  procedure  measures  of 
system  performance  on  a  logarithmic  ratio  basis  of  test  aircraft  configurations 
cerrpared  to  a  known  selected  standard  aircraft  (vehicle)  configuration.  The  logari¬ 
thmic  units  10  log  (^test/^std)  and  10  log  ^test/^std)  used  in  this  system 
evaluation  method  are  termed  ’  ciecivals,  dV*  as  they  are  10  times  the  log  base 
10  of  the  ratio  of  tlje  ECP  and  PRP  values  obtained  during  tests  for  the  test  air¬ 
craft  configuration foerrpared  to  the  cJosen  standard  aircraft  configuration.  ~-~ 


^~The  system  evaluation  is  for  chosen  time  periods  of  selected  flight  operations 
which  are  critical  to  flight  safety,  such  as  may  occur  during  takeoff,  or  approach 
to  landing  and  nay  include  emergency  engine  failure,  flight  control  or  instrument 
malfunction  conditions.  System  equations  are  presented  which  answer  the  question 
as  to  hew  good  is  the  test  configuration  in  relation  to  the  known  standard  con¬ 
figuration  during  these  sairejf light  conditions.  Potential  ECP  measures  are  dis¬ 
cussed  and  their  correlation  with  PRP  pilot  ratings  obtained  during  flight  test  or 
flight  simulator  test  determines  their  retention  as  effective  system  performance 
and  evaluation  measures.  The  non-dimensional  logarithmic  nature  of  the  retained  ECP 
system  perfomanoe  descriptors  allows  their  combination  by  logarithmic  summation 
and  their  correlation  with  the  PRP  pilot  ratings  is  determined.  The  ccmbinaticn  of 
ECP  measures  having  the  highest  correlation  with  pilot  ratings  is  retained  for 
final  system  evaluation.^— - - 
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A  complete  systems  engineering  method  is  presented  wh-thh  / 
provides  a  missing  frame  of  reference  and  a  procedure  including 
new  measurement  units  which  quantifies  and  simplifies  the  evalua¬ 
tion  of  piloted  aircraft  systems  tested  in  flight  or  the  systems 
evaluation  of  any  man  operated  vehicle  or  machine  tested  in  its 
operational  environment. 

New  pilot  rating  procedure s,  PRP,  and  new  engineering  cal¬ 
culation  procedures,  ECP,  ^re  presented  comparing  the  test  air¬ 
craft  configuration  ratings  and  measures  with  a  chosen  standard 
aircraft  configuration  using  non-dimensional  logarithmic  units 
termed  decivals,  dv. 

A  system  evaluation  equation  is  presented  which  quantifies 
and  describes  the  relationship  between  the  flight  test  pilot 
ratings  and  the  measured  flight  test  data.  This  method  may  be 
used  for  evaluation  purposes  during  system  design,  test,  certi¬ 
fication  of  airworthiness  and  flight  safety  evaluations  to  deter¬ 
mine  the  aircraft  in  flight  man-machine  pilot-aircraft  {operator 
vehicle)  operating  system  relationship: 


test 

PRPstd 


test 


ECPstd 


The  answer  to^  how  good  the  aircraft  or  vehicle  is,  regarding 
the  operationally  defined  system  evaluation  goal,  given  an 
engineering  calculation  procedure  ECP  measure  and^)  given  a 


pilot  or  operator  rating  procedure  PRP  rating  what  should  the 

|!CP,  measured  value  be  is  provided  by  the  above  equation  in 

\  * 

lbgaritlunic  form  as  follows: 


The  effectiveness  of  the  completed  system  evaluation  nuiy 
be  determined  by  the  product  moment  correlation  coefficient , 
*^PRP  .  ECP  obtained  and  ECp  showing  the  amount  of 

common  variance  between  pilot  ratings  and  flight  test  data. 
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INTRODUCTION 


A  system  is  defined  as  any  machine  or  tool  including  all 
modes  of  transportation  vehicles,  or  their  roadways,  aerospace- 
ways,  and  seaways,  or  signs  and  signals  used  by  man  to  attain 
some  goal.  All  systems  include  the  effects  of  the  .hanging 
man-made  or  natural  environment  in  which  they  are  used  and 
operated  in  real  life  and  real  time. 

There  are  therefore  a  great  many  extraordinarily  different 
kinds  of  systems,  all  of  these  systems  may  be  evaluated  to 
determine  exactly  how  useful  they  are  in  attaining  the  various 
goals  that  have  been  set  for  them.  It  can  be  seen  that  this 
depends  on  the  experience  or  ability  of  the  user  or  operator 
and  the  characteristics  of  the  machine  or  tool  or  vehicle  or 
roadway  or  sign  or  signal  and  the  changing  man-made  and  natural 
environment  in  which  it  is  used  or  operated  in  real  life  and 
real  time. 

There  is  therefore  a  great  need  for  an  objective  and  reli¬ 
able  method  of  evaluating  systems.  There  is  at  the  present 
time  no  complete  unifying  method  which  provides  a  frame  of 
reference,  and  which  defines,  quantifies  and  describes  the  re¬ 
lationship  between  the  criteria  for  evaluation  and  the  observable 
and  measureable  data  regarding  the  system  performance  in 
attaining  its  set  goal. 


Today  there  are  many  ways  to  describe  and  evaluate  the 
separate  and  component  parts  of  the  system.  For  example  there 
are  mathematical  models  of  the  human  operator  behavior  which 
are  summarized  and  explained  in  Reference  1  and  in  the  numerous 
references  cited  therein. 

There  are  many  excellent  rating  scales  such  as  the 
Cooper-Harper  pilot  rating  of  aircraft  handling  qualities 
described  in  Reference  2  and  its  references  lead  the  reader 
to  other  methods  available  for  obtaining  pilot  ratings (2)* 

A  summary  of  the  definition  and  measurement  of  perceptual 
and  mental  workload  is  contained  in  Reference  3  which  points 
out  that  the  common  method  of  assessing  handling  qualities 
relies  on  the  subjective  opinion  of  experienced  test  pilots <3) . 

Reference  1  emphasizes  that  these  subjective  pilot  opinions 
have  proved  to  be  quite  reliable  but  that  they  are  sometimes 
difficult  to  use  for  design  purposes.  It  also  stresses  that 
the  ultimate  goal  fn  building  systems  engineering  models  of 
operator  behavior  is  to  use  these  models  in  the  evaluation 
of  aircraft  handling  qualities  essential  for  flight  operations. 
Of  utmost  importance  is  the  acknowledgement  that  these 
handling  qualities  are  vital  for  flight  safety (D . 


But  there  is  a  severe  limitation  pointed  out  in  Reference  1 


regarding  these  engineering  models  which  have  been  postulated 
to  date,  quote: 

"A  rather  severe  limitation  of  both  engineering  models 
lies  in  the  fact  that  they  lack  a  direct  connection  with 
the  pilot's  subjective  opinion  of  the  workload  involved 
in  a  given  control  situation.  In  the  end  it  is  this 
expert  opinion,  which  decides  on  the  acceptability,  or 
otherwise,  of  an  aspect  of  the  aircraft's  handling  quali¬ 
ties.  There  exists,  as  yet,  no  generally  applicable 
method  to  derive  the  pilot's  workload  from  the  parameters 
of  the  engineering  model.  There  are  serious  doubts 
whether  such  a  method  will  ever  be  developed. 

An  explanation  of  this  very  real  shortcoming  of  most 
existing  engineering  models  is  considered  to  lie  in  the 
fact  that  such  models  portray  primarily  the  control 
aspect  of  the  human  operators  activities,  thereby  ignor¬ 
ing  or  bypassing  the  equally  important  mental  activities 
of  data  processing  and  decision  making It  seems, 
therefore,  that  there  is  a  need  for  new  developments, 
explicitly  combining  into  a  single  mathematical  model 
the  internal  mental  processes  of  data  handling  and 
decision  making  going  on  in  the  human  brain,  with  the 
more  overt  control  activities  previously  discussed. 


It  turns  out  to  be  a  truly  interdisciplinary  study 
for  which  an  accepted  frame  of  reference  has  not  yet 
been  established,  and  iteration  between  rather  general 
hypothesis,  test  of  methods,  and  detailed  analysis  is 
necessary. * 

This  report  is  intended  to  present  ideas  which  provide  the 
missing  frame  of  reference  for  the  total  system  description, 
analysis  and  evaluation.  This  is  accomplished  by  quantifying 
the  component  parts  of  a  system  as  defined  in  the  first  para¬ 
graph  of  the  introduction  and  the  degree  of  relationship  between 
the  test  user,  operator,  pilot,  driver  ratings,  and  the 
measured  test  system  input  and  output  performance  data. 

This  method  allows  the  system  evaluator  to  first  opera¬ 
tionally  define  his  system  evaluation  goals  and  then  conduct  a 
systems  evaluation  which  determines  how  good  are  the  measured 
test  data  as  well  as  how  good  are  the  test  user,  operator,  pilot 
driver  ratings.  Of  utmost  importance  is  the  fact  that  the 
method  allows  the  evaluator  to  determine  by  quantifying  it 
exactly  how  good  is  the  final  system  evaluation. 


PROCEDURE 


The  criteria  presented  for  determining  the  effectiveness 

of  a  completed  piloted  aircraft  system  evaluation  is  the  degree 

of  correlation  of  flight  test  pilot  ratings  PRP  with  engineering 
r 

calculations  procedure  ECP  measures  .  ECP.  The  higher  the 

«  product-moment  correlation  ^.ppp  .  the  better  the  engineer¬ 

ing  calculating  procedures  ECP,  selected  are  in  predicting  the 
combined  or  selected  system  characteristics  being  evaluated.  In 
like  manner,  the  higher  the  correlation  the  more  confidence 
that  may  be  placed  in  the  validity  and  reliability  of  the  flight 
test  pilot  (operator)  ratings. 

The  new  flight  test  pilot  rating  procedure,  PRP,  presented 
is  based  on  the  average  of  the  logarithms  of  the  pilot  (operator) 
magnitude  estimation  ratings  of  total  mental  and  physical  workload 
due  to  the  handling  qualities  and  performance  characteristics  of 
the  aircraft  (vehicle)  during  a  time  segment  of  a  selected  or 
required  critical  flight  (road,  sea,  space)  operation.  This 
rating,  PRP test  of  the  te8t  configuration  is  reference  a  given 
standard  aircraft  configuration  with  known  measured  handling 
qualities  and  performance  characteristics,  i.e.,  measured  and 
•  critical  flight  operation  which  is  given  a,  PRPgt^  pilot  mental 

and  physical  workload  rating  of  10.  If  the  test  aircraft  con¬ 
figuration  requires  twice  the  amount  of  the  pilots  mental  and 
physical  workload,  in  comparison  to  the  standard  configuration, 
the  pilot  would  rate  it  as  having  a  PRP  of  20.  If  one-half,  the 
pilot  would  rate  it  as  having  a  PRP test  of  5*  etc.  less 


than  one-half,  ratings  of  1  through  4  are  available  for  des¬ 
cribing  the  test  configuration  workload.  In  similar  fashion 
ratings  of  21  through  40  describe  workload  increasing  from 
double  to  four  times  the  selected  standard  workload. 

The  new  pilot  rating  procedure  is  therefore  stated 
mathematically  as  follows: 

PRP  *  10  log  (PRPtegt/PRPst(j) .  decivals,  dV.  (3) 

This  method  provides  a  pilot  rating  continuum  from  zero  or  a 
rating  of  the  test  configuration  as  being  identical  with  the 
standard  configuration  to  an  infinite  number  of  ratings. 

Ratings  in  a  positive  direction  indicate  a  doubling  of  the 
pilot-aircraft  system  workload  for  each  three  decival,  dV  in¬ 
crease.  The  negative  direction  indicates  a  halving  of  the 
pilot-aircraft  system  workload  for  each  three  decival,  dV 
decrease . 

The  new  engineering  calculation  procedures  presented  may 
be  stated  mathematically  as  follows: 

ECP  *  10  log  (ECPtest/ECPstd)  decivals,  dV.  (4) 

The  logarithmic  units,  10  log  (ECPtest/ECPst,3)  and  10  log 
(PRPj-ggt/PRPgta)  used  in  this  system  evaluation  method  are 
termed  "decivals,  dV"  as  they  are  10  times  the  log  base  10  of 
the  ratio  of  the  selected  ECP  and  PRP  system  evaluation  units 
for  the  test  aircraft  configuration  compared  to  the  chosen 
standard  aircraft  configuration. 


In  this  way  a  doubling  or  halving  of  the  ECP  or  PRP 
of  the  test  configuration  ratio  with  the  chosen  standard 
configuration  results  in  a  ±3  decival,  3  dv,  increase  or  de¬ 
crease  as  10  times  ldg  2  »  10  x  0.3  *  3  dV  and  10  log  (1/2)  - 
10  x  -0.3  -  -3  dV. 
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Each  ECP  is  selected  to  quantify  a  known  outstanding  cri- 
«  tical  characteristic  of  the  operating  system  such  as  integra¬ 

tion  of  the  pound-seconds  of  force  exerted  by  the  pilot  on  the 
flight  controls  about  che  pitch,  roll,  and  yaw  «.  ;is  of  the 
aircraft  during  the  time  segment  of  a  selected  flight  opera¬ 
tion  which  may  later  be  shown  to  correlate  highly  with  the 
pilots  ratings  of  physical  workload.  In  like  manner,  the  inte¬ 
gration  of  the  number  of  bits  of  information  transmitted  per 
second  by  the  primary  flight  instruments  during  the  time  segment 
of  the  selected  flight  operation  may  later  be  shown  to  correlate 
with  the  pilots  rating  of  mental  workload. 

Each  ECP  is  therefore  a  non-dimensional  logarithmic  ratio 
of  the  selected  critical  system  characteristic  ECP^0t^  to  the 
same  critical  system  characteristic  of  the  selected  standard 
aircraft  configuration  ECPstg.  This  is  a  very  important  part 
of  this  new  system  evaluation  procedure  as  it  provides  a  means 
*  of  converting  all  critical  systems  characteristics  to  non- 

dimensional  logarithmic  ratios  of  selected  critical  test  con¬ 
figuration  characteristics  compared  to  the  same  critical 
standard  configuration  characteristics  under  identical  test 
conditions.  Note  well  that  this  non-dimensional  nature  of 
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all  the  system  ECPs  permits  the  combination  of  all  selected 
critical  system  ECPs  into  a  single  integrated  ECP  for  the 
time  segment  of  the  selected  flight  operation.  This  is 
accomplished  by  summing  them  logarithmically: 

ECP  *  10  log  (antilog  ECPj/10  +  antilog  ECP2/IO  ....+  antilog  BCPj/10)  (5) 

It  is  important  to  note  that  pilot  ratings  PRP  are  also 
non-dimensional  as  they  are  10  times  the  log  of  the  ratio  of 
the  test  aircraft  configuration  to  the  selected  standard  con¬ 
figuration,  PRptest/pRPstd‘  Pilot  ratings  obtained  during  the 
same  time  segment  of  the  selected  flight  operation  for  both 
the  test  and  the  standard  aircraft  configurations,  therefore, 
provide  a  summary  PRP  test  pilot  rating  which  corresponds  to 
the  summary  ECP  flight  test  data  calculations.  It  is  the 
degree  of  correlation^  PRP  .  Ecp  of  these  values  obtained 
from  a  suitable  number  of  pilots  which  determines  how  effective 
the  selected  system  evaluation  ECP  measures  are  in  describing 
the  characteristics  of  the  operating  system  as  rated  by  the 
test  pilots. 

The  test  pilot  ratings  obtained  during  the  chosen  time 
period  of  the  selected  flight  operation  are  the  accepted 
criteria  for  determining  the  actual  real-life,  real-time 
characteristics  of  the  test  aircraft  in  its  flight  environment. 

As  the  PRP  and  ECP  measures  are  a  log  X  log  Y  relation¬ 
ship,  it  can  be  assumed  and  proven  subsequently  by  inspection 
that  their  relationship  is  rectilinear  and  that  a  straight 
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regression  line  equation  established  by  the  method  of  least 
squares  through  the  plotted  values  obtained  in  each  test 
flight  (or  flight  similator  test)  provides  the  answer  to  how 
good  is  the  aircraft  given  an  ECP: 


10  log  **  k  +  n  10  log 


(6) 


The  number  and  type  of  ECP  measures  chosen  for  an  evalu¬ 
ation  may  vary  from  one  measureable  characteristic  of  the 
piloted-aircraft  system  made  during  the  time  period  of  the 
selected  flight  operation  to  several  which  may  be  examined 
individually ,  in  pairs  and  all  possible  combinations.  The 
effectiveness  of  any  individual  ECP  measureable  piloted-aircraft 
system  characteristic  may  be  determined  by  its  degree  of 
correlation  with  the  PRP  flight  test  pilot  ratings  obtained. 
Those  with  low  correlations  may  be  rejected  while  those  which 
correlate  highly  may  be  retained  for  further  system  evaluation. 

The  goal  is  to  obtain  through  flight  and  simulator  test 
experience  those  ECP  measures  which  provide  the  highest  corre¬ 
lation  with  the  flight  test  pilot  ratings. 

There  are  two  significant  types  of  ECP  measures  available 
for  use  by  the  systems  engineer.  One  is  a  peak  measure  which 
is  the  maximum  value  obtained  by  integration  over 
the  complete  time  period  of  the  values  sampled  approximately 
every  0.5  or  1.0  second. 
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Previous  psychoacoustic  test  experience  has  shown  that 
the  time  integrated  measures  of  cockpit  sound  pressure  and 
frequency  provide  higher  correlation  with  pilot  ratings  of 
cockpit  perceived  sound  level  than  maximum  peak  measures.  The 
pilot  may i  therefore,  be  asked  to  rate  and  the  engineer 
measure  the  total  changing  situation  over  a  chosen  time  period 
of  a  selected  critical  flight  operation  as  well  as  the  peak 
value  occuring  during  that  time  period. 

Some  of  the  possible  measureable  ECP  characteristics  of 
the  piloted-aircraft  system  made  during  the  chosen  time  period 
of  the  selected  flight  operation  include  pilot-aircraft  flight 
control  induced  physical  workload;  pilot-aircraft  primary 
flight  instrument  display  induced  mental  workload;  total  in 
flight  system  pilot-aircraft  induced  mental  and  physical 
workload;  pilot-ATC  system-aircraft  navigation  communication 
display  and  control  induced  mental  and  physical  workload; 
pilot-aircraft  flight  operational  environment  induced  workload 
(turbulence,  wind  shear,  visibility,  IFR,  VFR,  Noise,  Vibration, 
Cockpit  lighting)  and  pilot-aircraft  induced  workload  due  to 
th«  aircraft  handling  quality-performance  descriptors  such 
as:  dutch-roll,  adverse  yaw,  longitudinal  stability,  lift 
over  drag  ratios  L/D  in  different  flap,  gear,  slot,  trim 
configurations,  etc.,  thrust  to  weight  ratios  T/w,  etc.  As 
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they  ultimately  become  available  with  the  operation  of  all 
systems,  the  aircraft  accident,  flight  safety  statistics  over 
time  for  the  various  critical  flight  operations  such  as  takeoff 
and  approach  to  landing,  etc.,  will  provide  the  most  signifi¬ 
cant  potential  ECP. 

The  degree  of  correlation  of  any  of  the  above  ECP  measures 
with  the  PRP  pilot  ratings  obtained  during  flight  tests  or 
flight  simulator  tests  for  a  time  period  of  a  selected  flight 
operation  determines  their  suitability  for  retention  and  con¬ 
sideration  in  the  final  ECP  measures  included  in  the  final 
system  evaluation. 

All  PRP  pilot  ratings  and  ECP  calculation  procedures  are 
logarithmic  ratios  between  the  test  aircraft  configuration  and 
the  chosen  standard  of  reference  for  all  ratings  and  measures, 
a  given  standard  aircraft  configuration,  during  a  time  period 
of  a  selected  flight  operation  flown  under  identical  or  as 
constant  as  practicable  test  conditions. 

All  flight  test  pilots  will  have  experience  and  sufficient 
training  in  the  selected  standard  aircraft  configuration  during 
the  time  period  of  the  selected  flight  operation  and  constant 
flight  test  conditions  to  assure  high  reliability  and  confidence 
in  their  ratings. 

All  system  evaluation  ECP  and  PRP  measures  are  calculated  in 
logarithmic  form  and  so  may  be  integrated  over  time  for  levels 
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sampled  at  discrete  time  intervals  of  for  example  (0.5) 
one-half  second.  The  working  expression  becomes: 


km±  PRP  /io 

/±t  (]() 

3EPRP  -  10  log  10  +  10  log  A* 

k^O 


K  ■  d 

At  ECP  /10 
— (k) 

1ECP  -  10  log  ^  10  +10  log  At 

k  «  0 


where  d  is  the  time  duration  in  seconds  during  which  ECP  (PRP)  is 
measured  and  At  is  the  time  interval  between  the  ECP  samples. 
Note  well  that  ECP^)  is  as  defined  by  Equation  Number  4,  i.e., 
10  log  <ECPte8t/ECPstd) . 

The  psychological  set  of  the  flight  test  pilots,  that  is 
the  operational  definition  of  the  system  evaluation  goals 
determines  the  outcome  of  the  evaluation.  The  psychological 
set  of  the  pilots  is  based  on  the  operational  definitions  used 
in  the  system  evaluation  which  define  precisely  what  is  being 
evaluated.  The  set  adopted  should  be  maintained  throughout 
for  rating  both  the  standard  and  the  test  aircraft  configura¬ 
tions.  The  possible  sets  include  but  are  not  limited  to  the 
following: 

(1)  The  whole  or  total  operationally  defined  pilot-aircraft 
system  workload  induced  by  the  aircraft  configuration 
during  a  chosen  time  period  of  a  selected  flight  opera¬ 
tion  such  as  an  ILS  or  MLS  approach  to  Dulles  Inter¬ 
national  Airport. 
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(2)  The  operationally  defined  physical  workload  only  asso¬ 
ciated  with  movement  of  the  flight  controls  and  inte¬ 
gration  over  time  of  the  pound-seconds  of  force  about 
the  pitch,  roll  and  yaw  axis. 

(3)  The  operationally  defined  mental  workload  only  associated 
with  the  primary  flight  instruments  and  integration 
over  time  of  the  bits  of  information  transmitted  per 
second  by  these  instruments. 

(4)  The  operationally  defined  mental  and  physical  workload 
of  (3)  and  (4)  above;  combined  in  decival,  dV  system 
evaluation  logarithmic  units. 

(5)  Other  psychological  sets  operationally  defined  according 
to  the  system  evaluation  goals.  For  example:  System 
evaluation  of  any  subsystem  change  such  as:  new  active 
flight  controls;  new  primary  flight  instruments  such 

as  electronic  altitude  director  indicator  changes;  ATC 
Communications-Navigation  changes;  new  collision  avoid-  , 
ance  or  data  link  changes,  etc.,  which  change  pilot 
mental  and  physical  maximum  peak  workload  or  workload 
over  time. 

The  degree  of  effectiveness  of  the  final  system  evaluation 
resulting  from  the  selection  of  appropriate  system  engineering 
calculation  procedure  ECP  measures  in  evaluating  the  piloted 
aircraft  or  any  human  operated  system's  performance  can  be  de¬ 
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termined  by  the  ECP  correlation  with  the  pilot  ratings,  i.e. 
by  the  value  of  the  positive  or  negative  product  moment 
correlation  coefficient  #  ECp  as  follows: 

TABLE  1.  INTERPRETATION  OF  SYSTEM  EVALUATION  RESULTS 


^PRP  .  ECP 

SYSTEM  EVALUATION  RESULTS 

a2 

PRP  .  Ei 

9b 

0.9  to  1.0 

Very  High 

81  to  100% 

0.78  to  0.89 

High 

61  to  80% 

0.64  to  0.77 

Moderate 

41  to  60% 

0.46  to  0.63 

Low 

21  to  40% 

0.00  to  0.45 

Very  Low 

0  to  20% 

♦The  amount  of  variance  in  test  pilot  ratings  PRP  that 
is  common  to  or  that  is  in  agreement  with  the  variance 
in  the  chosen  engineering  calculation  procedure  ECP 
test  data. 


A  detailed  explanation  of  the  product-moment  correlation 
coefficient  and  the  interpretation  of  correlation  upon  which 
the  above  Table  1  is  based  is  contained  in  Reference  6, 

Pages  79-106. 

An  example  of  system  evaluation  results  and  the  conclusions 
that  may  be  drawn  are  presented  in  Figure  1. 


An  example  of  the  major  steps  to  be  taken  in  conducting 
a  systems  evaluation  is  contained  in  Appendix  B. 
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.  AH  EXAMPLE  OP  SYSTEM  EVALUATION  RESULTS 


Pigure  1  is  an  exanple  of  syoten  svslustioa  rssults  and 
tha  following  is  an  exanple  of  conclusions  which  nay  be 
drawn! 

Tho  txptrt  flight  tut  pilot  rotingo,  P»  «Jtr*l«t»d 
highly^,,,,  ,  ^  -  0.10  with  tho  aouurod  flight  dot* 


Ibbbbbb 

Ibbbbbb 


This  result  shows  that  ovsr  80%  of  tha  variance  in  pilot 
ratings  PEP  is  associated  with  the  variance  in  the  flight 
test  data  ECP  which  indicates  a  very  high  qu*llt*tive_  _ 
•■•tea  evaluation  %  The  systen  evaluation  places  the  test 
configuration  as  being  twice  as  g©>*  °f 
one-half  the  pilot-aircraft  systen  lerkfaad  as  the  com¬ 
parison  standard  configuration  during  instrument  landing 
approaches  under  identical  flight  conditions. 
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The  chosen  or  'given'  standard  aircraft  configuration  is 
selected  for  comparison  purposes  of  both  the  pilot  or  operators 
ratings  and  the  flight  test  measures  obtained  for  the  test 
configuration.  If  the  comparison  or  standard  aircraft  con* 
figuration  is  one  with  which  the  test  pilots  are  already 
familiar  and  readily  available  for  flights,  this  simplifies 
both  the  choice  of  a  standard  and  the  problem  of  learning  its 
performance  characteristics. 

The  standard  configuration  is  not  presented  as  anything 
other  than  a  baseline  for  comparison  with  the  test  aircraft 
configuration  for  both  pilot  rating  PRP  and  engineering  calcu¬ 
lation  procedure  ECP  purposes.  It  provides  the  potential  means 
by  which  various  flight  test  data  as  different  as  apples  and 
oranges  are  turned  into  non-dimensional  logarithmic  ratio  unite 
termed  decivals  dV  and  thereby  these  at  first  seemingly  dif¬ 
ferent  types  of  measures  may  be  combined  by  logarithmic  summa¬ 
tion. 

This  provides  the  opportunity  to  ultimately  discover  by 
repeated  tests  and  exercise  of  this  system  evaluation  method 
the  most  important  ECP  measures.  These  measures  will  be  the 
£-um  of  the  flight  test  measured  data  parts  which  make  up  the 
.  uole  pilot  rating  of  the  aircraft  performance.  These  most 
;  portant  flight  test  measures  when  combined  have  the  highest 
correlation  with  the  test  pilot  ratings. 


Pilot  ratings  obtained  during  flight  tests  may  ba  ratings 
of  maximum  peak  workload  which  occurs  during  the  selected  flight 
segment  such  as  an  instrument  landing  system  ILS  approach.  It 
may  also  be  necessary  to  obtain  the  pilot  ratings  of  workload 
obtained  at  different  times  on  different  flight  segments.  In 
this  case  a  pilot  rating  of  total  workload  integrated  over  time 
may  be  computed  by  using  Equation  7. 

The  correlation  of  the  PRP  ratings  with  Cooper-Harper 
pilot  ratings  may  also  be  determined.  It  will  be  possible  to 
then  compare  the  Cooper-Harper  ratings  obtained  on  the  comparison 
or  standard  aircraft  with  those  for  the  test  aircraft.  Using 
the  logarithm  of  the  Cooper-Harper  ratings  will  permit  the 
checking  of  the  flight  teat  results  and  the  possibility  that 
the  present  1  through  10  rating  scale  is  somewhat  logarithmic. 

A  regression  line  established  by  the  method  of  least  squares 
will  quantify  the  log  linear  relationship  between  the  two  scales. 
Preliminary  paper  and  pencil  tests  (7)  show  that  their  is  a 
logarithmic  linear  relationship  between  the  pilots  or  rater's 
judgement  of  workload  doubling  and  the  Cooper-Harper  scale 
especially  in  the  mid-scale  4,  5,  and  6  ratings.  There  is  pre¬ 
sently  noc  enough  workload  described  by  ratings  near  the 
good  end  (present  ratings  of  l,  2,  and  3)  and  possibly  too  much 
at  the  bad  end  (7,  8,  9,  and  10)  to  meet  a  perfect  logarithmic 
linear  relationship  with  the  raters  judgement  of  the  rate  at 
which  workload  is  doubling  as  presently  described  by  the  1 
through  10  descriptive  language  associated  with  each  numerical 
rating.  (Test  (7)  results  are  contained  in  Appendix  A.) 
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This  could  be  achieved  by  psychometric  scaling  of  the 
descriptive  language  and  should  be  considered  in  future  modi- 
fixations  of  these  types  of  pilot  ratings  which  use  descrip¬ 
tive  language.  The  descriptive  language  in  each  of  the 
rating  choices  1  through  10  would  be  0.1  one-tenth  logarithmic 
distance,  i.e.  relative  rate  of  increasing  workload  as  follows: 

log  R:  0  .1  .2  .3. 4  .5  .6  .7  .8  .9  1.0 

R  :  1.  1.26  1.58  2.  2.5  3.16  4.0  5.0  o.31  8.00  10.0 

(Relative 
Rate  of 
Increasing 
Workload) 

Note  that  in  this  scale  the  workload  doubles  every  third 
descriptive  rating  of  workload  and  aircraft  handling  qualities. 

It  is  interesting  to  note  that  when  workload  has  increased 
10  times  over  some  minimum  or  ideal  workload  situation  that  it 
is  a  disaster  and  the  aircraft  is  uncontrollable.  Workload 
must  be  kept  at  levels  ideally  below  the  doubling  and  certainly 
the  quadrupling  of  the  minimum  or  ideal  workload  level  where 
R  *  1.0  and  log  R  =  0. 

This  emphasizes  the  need  to  increase  the  number  of  descrip¬ 
tive  ratings  presently  available  to  the  test  pilots  at  the 
good  or  low  workload  end  of  the  scale  and  the  development  of  a 
truly  logarithmic  descriptive  language  scaling  of  workload 
and  handling  qualities  of  aircraft. 


The  ideas  regarding  system  evaluation  and  pilot  ratings 
presented  here  are  intended  to  supplement  rather  than  replace 
the  present  Cooper-Harper  system  of  rating  aircraft  and  both 
types  of  pilot  ratings  would  be  obtained  as  well  as  other 
methods  not  discussed  herein  which  then  may  be  compared. 


In  fact  to  test  this  method,  the  following  may  be  carried 
out  not  only  for  future  tests  but  in  the  analysis  of  past 
recorded  flight  test  results  which  include  both  pilot  ratings 
and  measures  which  are  available  for  two  or  more  aircraft. 


The  decival  values  would  be  computed  for  the  pilot  ratings 
and  flight  test  measures  using  one  aircraft  chosen  arbitrarily 


as  the  comparison  or  standard  and  the  other  as  the  test  air¬ 


craft.  The  decival  value  of  10  log 


fc~Htest  "1  is 

c~Hstandardj 


determined 


for  each  test  pilot  and  compared  with  the  decival  value  of 
...  . ffiCP ...ii  . • ,  ■  .. «  »  .  * .  . 


10  log 


PV»rtest 

ECPstandard 


for  appropriate  measured  flight  test  data. 


The  degree  of  correlation  of  the  decival  pilot  ratings  with 


the  decival  flight  test  measures  will  aid  in  the  choice  of 
future  flight  teat  measurements  and  procedures. 


Where  changing  environmental  variables  are  of  concern 
as  to  their  effects  on  the  system  including  pilot  or  operator 
workload  and  the  handling  qualities  of  the  aircraft  or  vehicle 
then  the  test  variable  becomes  the  logarithmic  ratio  of  an 
environmental  change  with  time  compared  to  a  chosen  standard 
condition. 


The  decival  value  of  the  environmental  change  ENV  » 


10  log 


test  I  may  be  computed  and  compared  with  the  PRP 


decival  pilot  ratings  by  computing  their  correlation  coefficient* 
^*PRP  •  ENV,  obtained  for  these  same  environmental  test  and 


standard  conditions.  The  environmental  conditions  may  be  man 


made  or  natural  for  example:  changing  airport,  runway,  approach 
lighting  methods,  or  the  amount  of  turbulence  or  wind-shear 


for  repeated  ILS  approaches  by  the  same  piloted-aircraft.  For 
road  vehicles,  one  may  test  the  differences  in  roadways  and 
strive  to  learn  from  driver  ratings  and  handling  quality  mea¬ 
sures  why  there  are  more  accidents  for  left-hand  highway  curves 
compared  to  right-hand  curves.  Other  examples  of  conditions 
which  might  be  tested  in  this  way  include  the  effects  of  dif¬ 
ferent  kind  of  roadway  signs  and  signal  methods  as  well  as  the 


effects  of  weather  such  as  rain,  snow,  ice  on  the  road  safety 


of  the  national  highway  system. 


22 


Regarding  future  system  models ,  it  is  interesting  to  note 
that  the  relationship  between  pilot  response  and  system  output 
measures  having  high  correlation  with  each  other  based  on  the 
proposed  system  evaluation  method  may  be  used  in  developing 
and  improving  system  models. 

In  the  crossover/  optimum  control  and  other  system  models  ««• 
pilot  response  is  equal  to  10  log  |PRFtest]  decivals  and  the 

output  is  expressed  as  10  loo  lEcptestl  decivals. 

The  nature  of  the  units  of  pilot  response  and  system  output 
flowing  through  the  system  model  are  non-dimensional  logarithmic 
ratios  of  pilot  response  and  system  output  of  the  system  being 
modeled/  i.e./  the  test  system  compared  to  a  "given*  or  chosen 
standard  system.  This  represents  how  the  pilot  or  operator 
responds  to  the  displays  of  system  output  in  the  feedback 
loop.  He  is  comparing  how  the  new  or  test  aircraft/  vehicle/ 
machine  or  tool  responds  compared  to  a  known  or  learned  standard 
system  with  which  he  has  prior  experience.  This  is  suggestive 
of  an  approach  to  also  modeling  how  the  operator  learns  based 
on  his  experience  with  one  system  compared  to  another. 
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The  degree  of  proficiency  of  the  operators  of  systems 
being  evaluated  is  usually  a  problem.  In  the  present  method 
the  test  subjects  may  have  varying  degrees  of  proficiency 
as  they  will  be  rating  their  workload  on  a  test  system  relative 
to  a  standard  system.  Their  ratings  are,  therefore,  relative 
to  the  same  degree  of  proficiency  used  to  perform  the  selected 
operation  evaluated  for  both  the  standard  system  and  the  test 
system.  All  test  operator  ratings  and  test  measures  are  therefore 
equally  useful  for  system  evaluation  purposes  as  long  as  the 
degree  of  proficiency  is  such  that  the  test  operators  can 
perform  the  selected  operations  being  evaluated  from  begin¬ 
ning  to  end  on  the  test  and  standard  systems. 

Another  advantage  of  this  system  evaluation  method  is  that 
it  can  be  used  to  assure  that  new  evolving  systems  are  im¬ 
provements  over  previous  systems  as  to  their  handling  qualities 
and  operator  workload.  These  are  factors  which  have  a  direct 
bearing  on  safety.  The  improvement  or  departure  of  new 
systems  from  old  systems  with  known  records  of  safety  can 
easily  be  established  early  in  the  design  stage.  Continued 
use  of  this  method  will  identify  those  engineering  calculation 
procedures  or  measures  which  have  correlated  the  best  with  / 

past  system  performance  ratings.  These  will  be  the  best 
predictors  of  the  future  systems  performance.  Later  in  the 
design  process  the  method  may  be  used  in  simulation  tests  and 
continued  with  tests  of  the  prototype  and  first  operational 
vehicles. 
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A K  EXAMPLE  OF  A  SYSTEM  EVALUATION 


The  following  example  is  one  made-up  bv  the  author  to  demonstrate  the 
system  evaluation  procedure  more  clearly  than  mere  words.  Bach  step 
is  outlined  and  pilot  ratings  and  secured  data  were  selected  to 
demonstrate  an  evaluation  in  which  there  is  a  valid  relationship  be¬ 
tween  the  pilot  ratings  and  tho  test  data,  pound-seconds  of  force. 

This  is  an  exaaple  only  and  not  to  be  interpreted  as  a  valid  real- 
world  relationship. 

It  is  interesting  to  note  that  the  first  run  of  pilot  ratings  versus 

the  test  data  yielded  a  -0.39  correlation.  This  demonstrated  that  the 

\ 

first  selection  of  paired  ratings  and  test  data  was  indeed  a  random 
sample  and  there  was  no  significant  correlation  coefficient  or  rela¬ 
tionship  between  them.  This  may  be  the  outcome  of  a  real-world 
test  of  this  method.  But  it  is  equally  as  important  to  determine 
that  there  is  not  a  relationship  between  pilot  ratings  of  what  is 
being  evaluated  and  the  selected  test  data.  It  eliminates  that 
measured  system  variable  and  the  investigator  must  search  for  those 
measured  system  variables  which  do  correlate  with  the  pilots  ratings. 

As  these  significant  relationships  are  determined  they  nay  be  used  by 
everyone  and  system  evaluation  will  become  increasingly  simpler  and  more 
realistic  than  it  is  today. 
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AN  EXAMPLE  OF  A  SYSTEM  ENGINEERING  EVALUATION 


The  systems  engineering  evaluation  is  of  the  physical  work load/ hand ling 
qualities  of  a  simulator  versus  the  physical  workload/handling  qualities 
of  the  aircraft  simulated. 

Background  —  Because  of  undesirable  pilot  induced  oscillations 
found  in  the  first  simulator  model  the  control  forces  required  in  the 
simulator  during  an  approach  were  increased  by  the  designer  over  those 
found  in  the  aircraft.  This  evaluation  is  to  determine  first  whether 
this  difference  in  the  simulator  workload/handling  qualities  and  the 
simulated  aircraft  is  a  significant  variable  in  pilot  transition 
training  which  is  detected  by  the  pilots.  Of  equal  importance  is  to 
determine  if  pound- seconds  of  force  is  a  useful  measure  which  correlates 
highly  with  the  pilots  ratings  and  which  therefore,  may  be  used  as  a 
prediction  in  future  aircraft  design  and  evaluation  by  the  Federal 
Aviation  Administration  (FAA)  and  the  Industry. 

Procedure  — 

Step  1:  The  first  step  is  to  define  operationally  what  is  being 
evaluated  which  sets  the  system  engineering  evaluation  goals. 

System  engineering  evaluation  goal:  Do  the  control  forces  in  the 
simulator  affect  pilot  transition  training  to  the  simulated 
aircraft?  Specifically  are  the  increased  control  forces,  measured 
in  pound-seconds  integrated  over  time,  required  in  the  simulator 
to  avoid  pilot  induced  oscillations  during  a  standard  instrument 
approach  noticed  and  rated  as  such  by  the  pilots  after  they  have 
successfully  transitioned  to  the  aircraft  and  obtained  actual 
flight  experience?  Operationally  defined  the  question  is  as 

follows:  What  are  the  pilot  ratings  of  physical  work- 
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load  in  the  simulator  during  a  standard  instrument  approach  compared 
to  a  given  physical  workload  rating  of  10  in  the  aircraft?  And  most 
isqwrtant:  Is  there  a  correlation  between  the  chosen  measure,  total 
pound-seconds  of  force  integrated  over  tim&and  the  pilot  ratings 
of  physical  workload? 

Step  2;  Select  the  aircraft  standard  configuration- -This  is  the 
real  aircraft  configuration  in  flightduring  a  standard  instrument 
approach.  The  standard  is  smlectedfor  comparison  purposes  only. 

It  is  the  configuration  which  has  been  incorporated  in  the  simu¬ 
lator  and  it  is  the  one  which  the  pilots  are  most  familiar  with 
and  obtained  flight  experienced  in  and  is  readily  available  for 
flight. 

Step  3:  Select  the  flight  test  engineering  calculation  procedure 
ECP,  measures  that  will  be  recorded:  The  pound-seconds  of  force 
required  about  the  pitch,  roll  and  yaw  axes  will  be  integrated  over 
time  to  obtain  objective  measures  of  the  forces  actually  required: 

(1)  To  fly  the  selected  aircraft  standard  configuration  in  actual 
flight  and  (2)  in  the  simulator  from  the  outer  marker  to  touch¬ 
down  and  roll-out  during  a  standard  instrument  approach  to  the 
local  airport. 

Step  4:  Train  the  pilots  so  that  they  understand  what  is  being 
rated  as  defined  in  Step  1:  What  are  the  magnitude  estimation 
pilot  ratings  of  physical  workload  in  the  simulator  during  a 
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standard  instrument  approach  compared  to  a  given  physical  workload 
rating  of  ten  in  the  aircraft  standard  configuration  selected  in 
Step  2  and  flown  under  similar  conditions  on  a  standard  instrument 
approach?  Pilots  are  instructed  to  rate  the  simulator  physical 
workload  as  five  if  one-half  that  of  the  aircraft  physical  workload 
and  20  if  doubled  with  any  rating  from  zero  to  infinity  being  avail¬ 
able  to  quantify  their  rating  of  simulator  physical  workload  in 
relation  to  the  selected  standard  aircraft  configuration  physical 
workload  given  a  rating  of  ten. 

Step  S:  Ten  pilots  fly  the  selected  standard  aircraft  configuration 
and  learn  during  three  instrument  approaches  that  the  operationally 
defined  physical  workload  required  to  fly  the  standard  instrument 
approach  from  the  outer  marker  to  touchdown  and  roll  out  is  given 
a  magnitude  estimation  value  of  ten. 

Step  6:  The  same  ten  pilots  fly  three  identical  instrument  approaches 
in  the  simulator  which  is  being  evaluated.  Bach  pilot  provides  his 
magnitude  estimation  judgement  of  the  physical  workload  required 
to  fly  the  flight  simulator  with  its  known  simulated  aircraft 
characteristics  from  the  outer  marker  to  touchdown  and  rollout 
compared  to  the  physical  workload  required  to  fly  the  actual 
aircraft  under  identical  conditions  given  a  physical  workload  rating 
of  ten.  If  in  the  pilot's  judgment  the  physical  workload  in  the 
simulator  is  double  that  required  by  him  to  fly  the  actual  approaches 
his  rating  is  20.  If  one-half  his  rating  is  five.  Note  well:  The 
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pilot  My  use  any  rating  fro®  zero  to  infinity  to  describe  and 
aost  important  quantify  their  ratings  on  a  continuous  scale 
which  is  anchored  to  a  known  and  quantified  standard  physical 
workload , 

Step  7:  Record  the  selected  test  data  during  the  actual  flights 
and  the  sioulated  approaches  so  that  they  My  be  quantified  (ECP) 
and  compared  to  the  pilot  ratings  (PRP)  and  their  exact  relation¬ 
ship  to  each  other  determined,  i.e.,  ^ PRP  .  ECP  will  equal  some 
number  between  -  1.0  and  1.0  which  quantifies  the  degree  of 
relationship  to  each  other  determined  between  the  selected  and 
recorded  test  data  (ECP)  and  the  pilot  judgements  (PRP). 

The  selected  test  data  recorded  on  all  approaches  ir.  the  aircraft 
and  simulator  are  the  total  pound-seconds  of  force,  measured  by 
electrical  transducers,  about  the  pitch,  roll  and  yaw  axes  inte¬ 
grated  over  time  resulting  from  the  pilots  control  movements 
from  the  outer  Mrker  to  touchdown  and  rollout. 

Step  8:  Compute  for  each  flight  and  each  simulated  approach  the 
total  pound- seconds  of  force  measured  about  the  three  axes  from 
the  outer  marker  to  touchdown  and  rollout.  Determine  (1)  the 
total  pound-seconds  of  force  for  each  pilot  during  the  three 
actual  flights  and  (2)  the  total  for  each  pilot  for  the  three 
flights  in  the  simulator,  (3)  the  average  total  pound-seconds  of 
force  for  each  pilot  and  the  ratios  of  the  force  required  in  the 
simulator  cor r***eJ  to  that  required  in  the  aircraft. 
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Force  (pound-seconds) 


Simulator 


Force  (pound-seconds) 

Aircraft 

(4)  Calculate  the  logarithm  of  this  ratio  and  multiply  by 
ten  to  obtain  the  ECP  value. 


ECP  •  10  log1Q 

The  ECP  is  the  Engineering  Calculation  Procedure  used  to  obtain  a 
logarithmic  continuous  non-dimensional  measure.  (See  example,  Table  1) 


Force  (pound-seconds) 

Simulator 


Force  (pound-seconds) 

Aircraft 


Step  9:  In  a  similar  fashion  to  Step  8  (1)  Record  for  each 
simulator  test  flight,  the  pilot’s  magnitude  estimation  of  the 
physical  workload  required  to  fly  the  simulator  (2)  determine 
for  each  pilot  the  ratio  of  these  magnitude  estimation  ratings 
compared  to  the  aircraft’s  given  magnitude  estimation  value  of 
10. 


Magnitude  '  stimation  Rating 

(0  to  00)  _ _ Simulator 


Given  Magnitude  Estimation  Rating 
(10)  Aircraft 


(3)  Calculate  the  logarithm  of  this  ratio  and  multiply  by  10. 
This  is  the  pilot  rating,  PRP  value  entered  in  the  last 


column  of  Table  2  for  each  pilot  A  through  J. 

PRP  «=  10  log10  fM-E-  Ratingsiioulator/Given  M’E*  ^ting  Aircraft) 
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AN  EXAMPLE  OF  SYSTEM  ENGINEERING  EVALUATION  TEST  DATA  AND  RESULTS 

i  1 . . . . 1  '■■■“■'  . -  * 

i 

Engineering  Calculation  Procedures,  ECP  —  The  following  data  continued 
in  Table  1  are  the  measured  test  data  obtained  from  electrical  transducers 
which  measured  the  total  pound-seconds  of  force  required  about  the  pitch, 

i 

roll  and  yaw  axes  recorded  during  simulator  and  aircraft  flights  from 
the  outer  marker  to  touchdown  and  roll  out. 


Table  1  shows  how  the  measured  data  are  averaged  in  both  the  simulator 
and  aircraft  and  their  ratios  obtained,  simulator:  aircraft,  and 
converted  into  10  log  form  to  obtain  ECP  values. 
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Pilot  Itatlm  Procedure,  PRP  —  Tho  following  data  contained  in  Table  2. 
are  the  pilot  magnitude  estimation  ratings  of  pilot  physical  workload 
coopered  to  aircraft  physical  workload  given  a  magnitude  estloation 
rating  of  ten.  Ratings  are  obtained  fron  the  sane  tea  pilots  listed 
in  Table  1  as  pilots  A  through  J. 

Tho  ratio  of  the  average  aagnitude  estimation  of  physical  workload  in 
the  simulator  for  each  pilot  compared  to  the  aagnitude  estimation  of 
physical  workload  in  the  aircraft  given  a  value  of  ten  is  calculated 
for  each  pilot.  Ten  times  the  logarithm  of  this  average  ratio£  10  log 
(average  ratio)J gives  the  pilot  rating  procedure ,  PRP,  value  for  each 
pilot  A  through  J. 
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TABLE  2.  PILOT  MAGNITUDE  ESTIMATION  RATINGS  OP  SIMULATOR 
PHYSICAL  WORKLOAD  COMPARED  TO  AIRCRAFT  WORKLOAD 
GIVEN  A  MAGNITUDE  ESTIMATION  RATING  OF  TEN 


An  alternative  quantified  descriptive  system  perfcjownce/opsarator,  pilot  or 
controller  workload  ^ total  mental  and  physical  effort;  rating  procedure  is 
presented  in  Appendix  C. 
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The  relationship  or  correlation  between  pilot  ratings,  PRP  and  the 
engineering  measures,  ECP,  may  now  be  obtained.  From  this  quantified 
relationship,  two  equations  based  on  the  least  squares  fit  of  the 
PRP  and  ECP  data  nay  be  obtained  so  as  to  predict  one  of  these  measures 
based  on  the  other. 

The  magnitude  of  the  relationship  may  be  compared  to  a  table  of 
correlation  coefficients,  such  as  Table  0  in  Reference  6. 

According  to  the  number  of  degrees  of  freedom,  N  pairs-lsnine  in  the 
present  case,  correlations  larger  than  0.602  and  0.735  could  occur 
by  chance  and  chance  alone  only  five  and  one  time  in  a  hundred  re¬ 
spectively. 

Table  5  presents  an  example  of  this  process. 
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TABLE  3.  THE  TEST  OF  RELATIONSHIP  OF  MEASURED  TEST  DATA, 

ECP  WITH  PILOT  RATINGS,  PRP 


NUMBER  OF 
AVERAGED 

PAIRS  PRP  ECP  RESULTS 


1 

2 

3 

4 

5 

6 

7 

8 


2.553 
2.3 
3.424 

2.553 
2.3 
3.424 
3.22 
3.01 


3.01  H 

PRP  .  ECP  ■  0.89* 

2.553  The  correlation  of  pilot  ratings  with  measured 
flight  test  date. 

3.29 

Eq.l.  PRP  ■  0.647  4  1.161  (ECP) 

3.01  Used  to  predict  pilot  ratings,  PRP,  based  on 
■ensured  flight  test  data,  ECP. 

2.34 

Bq.2.  ECP  -  1.07  4  0.68  (PRP) 

3.554  Use  to  predict  Measured  flight  test  data,  ECP 
based  on  pilot  ratings,  PRP. 

3.144  ^  o 

^  2  m  0.89*  -  80* 

3.01  CoMon  variance  in  ratings  and  flight  test  data. 


9  3.22  3.29 

*  Statistical  tables  such  as  Table  D  of 
10  2.3  2.7S  Reference  6  show  that  for  10  pairs  of  data 

the  correlation  aust  be  greater  than  0.735 
to  be  99  percent  confident  that  the  correlation 
is  real  and  did  not  occur  by  chance  and  chance 
alone.  Because  of  the  0.89  correlation  the 
systea  evaluation  is  valid  and  Equation  1  will 
be  useful  to  predict  pilot  ratings  of  physical 
workload  based  on  aeasured  or  calculated 
total  pound- seconds  of  force  values. 


CONCLUSIONS 


A  system  evaluation  method  is  presented  which  systematizes 
and  quantifies  both  PRP  pilot  rating  procedures  and  ECP 
engineering  calculation  Procedure  measures  of  system  performance 
on  a  logarithmic  ratio  basis  of  test  aircraft  configurations 
compared  to  a  known  selected  standard  aircraft  (vehicle)  con¬ 
figuration.  The  logarithmic  units  10  log  (ECPtest/ECPstd)  and 
10  log  (PR^test^^std^  used  in  this  system  evaluation  method 
are  termed  "decivals,  dV"  as  they  are  10  times  the  log  base 
10  of  the  ratio  of  the  ECP  and  PPP  values  obtained  during 
teats  for  the  test  aircraft  configuration  compared  to  the 
chosen  standard  aircraft  configuration.  v 

Tha  system  evaluation  is  for  chosen  time  periods  of 
selected  flight  operations  which  are  critical  to  flight  safety, 
such  as  may  occur  during  takeoff,  or  approach  to  landing  and 
may  include  emergency  engine  failure,  flight  control  or  instru¬ 
ment  malfunction  conditions.  System  equations  are  presented 
which  answer  the  question  as  to  how  good  is  the  test  configura¬ 
tion  in  relation  to  the  known  standard  configuration  during 
these  same  flight  conditions.  Potential  ECP  measures  are 
discussed  and  their  correlation  with  PRP  pilot  ratings  obtained 
during  flight  test  or  flight  simulator  test  determines  their 
retention  as  effective  system  performance  and  evaluation 
measures.  The  non-dimensional  logarithmic  nature  of  the  rc— 
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tained  ECP  system  performance  descriptors  allows  their  com¬ 
bination  by  logarithmic  summation  and  their  correlation  with 
the  PRP  pilot  ratings  is  determined.  The  combination  of  ECP 
measures  having  the  highest  correlation  with  pilot  ratings 
is  retained  for  final  system  evaluation. 
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APPENDIX 


Two  tests  were  conducted  where  32  subjects,  11  pilots, 
and  21  non-pilots  rated  the  increasing  workload  based  on  the 
Cooper-Harper  descriptive  language  regarding  aircraft 
characteristics, demands  oi»  the  pilot,  and  adequacy  of  the  air¬ 
craft  for  selected  task  or  required  operation.  The  required 
operation  was  limited  to  takeoff,  landing  or  cruise  under 
visual  flight  rules  and  non-turbulent  weather. 

In  test  one,  the  32  subjects  estimated  the  increasing 
workload  for  each  Cooper-Harper  pilot  rating  one  through  10 
reference  a  standard  aircraft  with  a  pilot  rating  of  one,  i.e., 
a  C-H  rating  of  one.  This  C-H  rating  number  one  was  given  a 
standard  workload  rating  of  10.  Where  the  aircraft  character¬ 
istics  and  demands  on  the  pilot  are  doubled  the  subjects  gave 
that  C-H  number  a  rating  of  20.  When  that  workload  doubled 
they  entered  40  and  80  for  the  next  doubling,  etc. 

In  test  two,  conducted  60-days  later,  the  32  subjects 
rated  the  increasing  and  decreasing  workload  but  Cooper- 
Harper  pilot  rating  6  was  given  a  workload  standard  rating 
of  10.  The  test  subjects  rated  as  in  test  one  but  also  where 
workload  was  one-half  they  entered  5  and  one-half  of  that 
workload  they  entered  2.5,  etc. 
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APPENDIX  (Cont’d) 


The  hypothesis  tested  end  the  results  of  the  tests  were 
as  follows! 

C-H  Sating! 


1  2  3  4  5  6 

Hypothesis  Tested  (Log  workload  Ratio) ! 

7 

8 

9 

10 

0  0.1  0.2  0.3  0.4 

Test  1  (Log  Workload  Ratio) i 

0.5 

0.6 

0.7 

0.8 

0.9 

c  0  .  .  0.04  0.176  0.3  0.43 

Standard 

Test  2  (Log  Workload  Ratio) ! 

0.57 

0.76 

0.92 

1.1 

1.25 

“0.04  -0.16  0.11  0.3  0.462 

Hypothesis  Tested  (Workload  Ratio)! 

0.57  0.84 

Standard 

1.0 

1.28 

1.55 

1  1.25  1.58  2  2.5 

3.16 

4. 

5. 

6.5 

8.0 

Test  1  (Workload  Ratio) ! 

1  1.1  1.5  2  2.7 

Standard 

3.7 

X 

5.7 

8.3 

12.7 

18.0 

Test  2  (Workload  Ratio) ! 

0.37  0.65  1.3  2  2.9 

3.7 

6.9 

10.4 

19.0 

35.5 

Standard 


The  rating  of  the  pilots  was  not  significantly  different  from 
the  non-pilots. 
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APPENDIX  B 


Steps  in  Systems  Evaluation 

1.  Set  the  system  evaluation  goals:  Operationally  define  what 

v 

is  being  rated.  (See  Page  14,  Section  (1)  through  Section  (5) 
for  examples.) 

2.  Select  the  aircraft  standard  configuration  for  comparison  pur- 
poses  only.  It  is  one  with  which  the  raters  are  most  familiar 
and  is  readily  available  for  flight.  (See  Page  18  for  dis¬ 
cussion.) 

3.  Train  the  raters  regarding  what  is  being  rated  and  what  the 
system  evaluation  goal  is  using  operational  definitions  and 
the  magnitude  estimation  technique  reference  a  standard 
workload  (or  what  is  being  evaluated  as  defined  in  Step  1) 

of  10  for  the  comparison  system.  (See  Page  14  for  discussion.) 

4.  Select  the  flight  test  data  ECP  measures  that  will  be  moni¬ 
tored  and  recorded  regarding  test  and  standard  system  per¬ 
formance.  (See  Page  9  for  discussion  of  ECP  examples.) 

5.  Raters  fly  the  selected  standard  aircraft  on  the  chosen  time 
segment  of  a  required  flight  operation  such  as  an  ILS 
approach  and  learn  that  the  operationally  defined  workload 
(or  what  is  being  evaluated  as  defined  in  Step  1)  is  equal 
to  10. 

6.  Raters  fly  the  test  aircraft  on  the  same  time  segment  of  the 
chosen  flight  operation  such  as  an  ILS  approach  and  obtain 
pilot  ratings  of  the  operationally  defined  workload  (or 
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what  is  being  rated)  set  in  Step  1.  Use  magnitude  estima¬ 
tion  to  judge  how  workload  compares  to  the  arbitrarily 
chosen  but  learned  standard.  If  double,  rate  it  20,  if 
one-half  rate  it  5,  etc.  (See  Page  7  for  discussion.) 

7.  Record  selected  flight  test  data,  ECP  measures,  during  the 
operation  of  both  the  test  and  standard  aircraft  for  which 
pilot  ratings,  PRP  magnitude  estimates  of  workload,  are 
obtained  and  a  standard  operationally  defined  workload  (or 
what  is  being  rated  as  net  in  Step  1)  of  10  set  for  the 
standard  aircraft. 


8.  Compute  pilot  magnitude  estimates  of  workload: 

PRP  ■  10  log  p^testl  in  decivals,  dV. 

Li<wj 

9.  Compute  system  flight  test  data  measures: 

ECP  ■  10  log  f*ECPtestl  in  decivals,  dV  for  each  selected 


important  system  variable  selected  in  Step  4. 

10.  Determine  the  quantified  relationship  ranging  from  sero  to 
plus  or  minus  1,  i.e.  the  product-moment  correlation 
coefficient^  PRP  .  ECP  between  all  the  pilot  ratings  deter¬ 
mined  in  Step  8  with  the  corresponding  flight  test  data 
measures  selected  in  Step  9.  (See  Reference  6  for  detailed 
explanation  of  the  correlation  coefficient.) 
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11.  Retain  those  measures  which  correlate  highly  with  the  pilot 
ratings.  Discard  those  which  lack  correlation.  Combine 

by  logarithmic  summation  those  measures  which  correlated 
best  with  pilot  ratings,  (independent  ECP  measures) 

12.  Compute  the  correlation  between  the  combined  ECP  measures 
retained  and  pilot  ratings  where  “^Tprp  .  ECP  equals  +1.0 
is  perfect  correlation  and  zero  is  none.  (See  Page  16 
for  discussion  of  interpretation  of'^’PRP  .  ECP  and  system 
evaluation  results.) 

This  is  the  degree  of  effectiveness  of  the  system  evaluation 
and  provides  a  check  on  the  predictive  value  of  the  flight 
test  data  as  well  as  the  reliability  of  the  pilot  ratings* 

PRP.  As  more  system  evaluations  using  the  method  are  com¬ 
pleted  there  will  be  a  growing  identification  of  which  flight 
test  data  measures*  ECP*  are  the  best  predictors  of  the  opera¬ 
tionally  defined  system  evaluation  goals  set  in  Step  1. 
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APPENDIX  C 


item  Perf ccRerce/Operator ,  Pilot,  and  Controller 


The  addition  of  mathematical,  operational  definitiona  of  workload  and 
ay a tea  performance  language  to  quantify  preciaely  and  denote  exactly 
the  incraaaing  and  decreaaing  rate  aaaociated  with  the  preaent  (revised) 
Cooper-Harper  (!)*  rating  scale  language  is  suggested  as  a  way  to  aid  the 
test  pilot  in  rating  aircraft.  This  will  also  aid  the  aviation  community, 
users,  designers,  systems  analysts,  scientists,  engineers,  etc,,  in  better 
understanding  of  what  the  rating  means. 

It  is  suggested,  based  on  tests  and  ideas  contained  in  reference  (2),  that 
by  definition,  the  first  Cooper-Harper  rating  of  1.0  be  assigned  a  workload 
(standard)  ratio  of  1.0.  The  workload,  again  by  definition,  associated  with 
the  Cooper-Harper  rating  of  4.0  is  now  assigned  a  workload  ratio  (compared  to 
the  standard)  of  2.0  or  a  doubling  of  workload  due  to  the  language  presently 
assigned  to  the  Cooper-Harper  rating.  Again  by  definition,  the  workload 
aasociatad  with  a  Cooper-Harper  rating  of  7.0  is  assigned  a  workload  ratio 
(compared  to  the  standard)  of  4.0.  This  is  a  doubling  of  the  workload 
aaaociated  with  the  change  in  language  from  a  Cooper-Harper  rating  of 
4.0  to  one  of  7.0.  In  aimilar  fashion  by  definition  the  workload  asso¬ 
ciated  with  the  language  of  a  Cooper-Harper  rating  of  10  is  assigned  a 
workload  ratio  (compared  to  the  standard)  of  8.0.  This  is  a  doubling  of 
the  workload  associated  with  a  change  in  the  language  from  a  Cooper- 
Harper  rating  of  7.0  to  one  of  10. 

This  procedure  has  by  definition  fixed  the  rate  of  doubling  of  workload/ 
system  performance  every  third  block  starting  from  the  first  Cooper- 
Harper  rating  of  1.0.  This  gives  us  at  this  point  the  following  exact 

*Denotes  reference  number 


46 


definition  of  workload  and  it*  rate  of  increase/decreaae  associated  with 
the  present  language  of  the  Cooper-Harper  rating  scale: 


Cooper-Harper  Rating: 

1  2 

3  4 

5  6  7  8 

9  10 

Workload  Ratio: 

1.0 

2.0 

4.0 

8.0 

The  Log  Workload  Ratio: 

0 

0.3 

0.6 

0.9 

It  only  remains  at  this  point  to  perceive  that  this  defines  and  assigns 
quantified  workload  values  to  the  present  Cooper-Harper  rating  scale  in 
an  easily  understood  fashion,  the  doubling  of  workload  every  third  rating. 
This  by  happy  accident  (serendipity)  is  a  logarithmic  scale  and,  therefore, 
the  intervening  blocks  are  also  defined  to  be  logarithmic  which  establishes 
for  the  complete  Cooper-Harper  rating  scale  the  rate  at  which  pilot  mental 
and  physical  workload  varies  with  the  language  associated  with  each 
rating. 

The  workload  for  the  Cooper-Harper  scale  is,  therefore,  defined  as  follows: 

Cooper-Harper  Rating:  123456789  10 

Workload  Ratio:  S.O  1.25  1.6  2.0  2.5  3.2  4.0  5.0  6.4  8.0 

The  Log  Workload  Ratio:  0  0.1  0.2  0.3  0.4  0.5  0.6  0.7  0.8  0.9 

The  defined  workload  ratio  associated  with  each  rating  is  seen  to  double 
with  every  third  rating  as  follows: 

Cooper-Harper  Rating:  123456789  10 

1.0  1.25  1.6  2.0  2.5  3.2  4.0  5.0 


L . I 


Workload  Ratio: 


6.4 


8.0 


This  quantified  definition  of  workload,  therefore,  simplifies  the  rating 
task  for  the  test  pilot  raters,  i.e. ,  the  doubling  or  halving  of  workload 
in  relation  to  the  language  contained  in  each  rating  block  every  third 
block.  As  a  result,  the  increasing  aircraft  system  evaluation  rating 
scale  is  logarithmic.  The  distances  between  the  ratings  are  now,  by 
definition,  equal  to  one-tenth  (0.1)  logarithmic  units  apart. 

The  workload  ratios  from  rating  block  to  rating  block  can,  therefore, 
be  precisely  defined.  The  workload  ratios  are,  of  course,  the  antilogs 
or  the  numbers  whose  logarithm  defines  the  amount  of  workload  associated 
with  each  rating  block. 

This  quantifying  of  the  workload  associated  with  each  pilot  rating  not 
only  simplifies  the  rating  task  for  the  pilot  but  perhaps  most  importantly 
helps  to  quantify  and  therefore  simplify  the  analysis  of  the  data  asso¬ 
ciated  with  many  pilots  and  many  flights,  etc. 

Revised  quantified  rating  forms  for  aircraft,  air  traffic  control,  and 

systems  evaluation  in  general  are  contained  in  the  Appendix. 

<* 

Uorkload  for  systems  in  general  is  defined  as  the  total  work  associated 
with  all  mental  and  physical  tasks  per  unit  time  period  required  to 
successfully  complete  the  selected  task  or  required  operation  under  a 
given  system  configuration  and  operating  conditions. 

In  each  case,  definition  of  the  required  operation  involves  designation  of 
the  system  phase  and/or  subphase  with  accompanying  conditions. 


The  pilot,  sir  traffic  controller,  or  system  operator  ratings  are 
arbitrarily  called  decivala  (dV).  A  decival  (dV)  is  equal  to  10  log1Q 
(quantified  workload  ratio).  The  decival  is  selected  rather  than 
decibel  (dB)  to  clearly  distinguish  it  from  the  reference  level  used 
in  electrical  engineering  practice.  The  decival  say  also  be  used  in 
syatea(s)  evaluation.  The  reference  level  is  defined  by  the  language 
associated  with  the  system  in  the  first  rating  block  which  has  a 
defined  value  of  1.0. 

Workload  Ratio: 

Log 
dV 

DISCUSSION 

The  quantified  system  performance /pilot,  controller,  operator  workload 
rating  scale  provides  a  continuum  of  possible  ratings  ranging  from  0  to  9 
decivals.  This  is  a  continuum  ranging  in  terms  of  system  performance/ 
workload  ratios  from  the  baseline  condition  of  1/1  or  1  to  8/1,  8  times 
or  3  doublings  of  the  defined  baseline  condition.  The  pilots,  air  traffic 
controllers,  or  operators  of  any  system  may  make  magnitude  estimate  judge¬ 
ments  of  the  system  performance /workload  anchored  not  only  at  the  baseline 
but  at  e  total  of  10  points  by  the  language  of  reference  (1)  already  in 
wide  use  to  rate  the  handling  qualities  of  aircraft. 


1.0  1.23  1.6  2.0  2.5  3.2  4.0  5.0  6.4 

0  .1  .2  .3  .4  .5  .6  .7  .8 

0  1  2  3  4  5  6  7  8 
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The  quantified  continuous  nature  of  the  eeale  providee  both  the  rater 
and  the  scientist,  engineer,  systems  analyst,  etc.,  with  an  infinite 
number  of  possible  ratings.  These  ratings  may  not  only  be  located  at  the 
10  points  defined  by  the  language  describing  the  system  performance/work¬ 
load  but  at  any  point  in  between  these  points. 

Pox  example,  the  rater  and  system  analyst  finally  have  a  way  of  deter¬ 
mining  where  a  system  is  precisely  which  may  be  between  the  third  and 
fourth  rating  Choices  in  system  performance /workload.  The  rater  ia  not 
limited,  for  example,  to  a  judgement  of  either  1.6  or  2.0  times  the 
quantified  workload.  The  rater  may  select  these  values  which  are  defined 
by  the  language  of  reference  (1)  or  may  interpolate  to  any  value  in  between 
to  deacribs  his  exact  magnitude  estimate  rating  of  the  system  performance/ 
workload. 

C0HCLUSI0NS 

Three  quantified  continuous  system  performance /workload  rating  scales  are 
presented  for  use  in  obtaining  pilot,  sir  traffic  controller  ratings  of 
aircraft  and  air  traffic  control  systems,  and  user/operator  ratings  of 
any  system. 

The  addition  of  mathematical,  operational  definitions  of  system  performance/ 
workload  language  is  suggested  to  quantify  precisely  the  exact  ratio  associated 
with  the  present  (revised)  Cooper-Harper  rating  scale  language  as  a  way  to 
aid  test  pilots,  air  traffic  controllers,  etc.,  in  rating  aircraft  and  air 
traffic  control  systems.  This  approach  simplifies  analyses  and  improves 
the  understanding  of  the  meaning  of  averages  of  pooled  raf ings  of  system 
operators/users . 
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Afppmix 


The  Three  Fora* 

1  ATC 

2  Aircraft 


3  General 


*  Otclval  Id')  Is  equal  to  10  le|..  (l)watUM  noitlMl  tatlo 


